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Highlights 

Highlights  of  GAO-1 0-774,  a  report  to  the 
Committee  on  Armed  Services,  U.S. 
Senate 


Why  GAO  Did  This  Study 

In  May  2009,  Congress  passed  the 
Weapon  Systems  Acquisition 
Reform  Act  of  2009  (Reform  Act). 
The  Reform  Act  contains  a  number 
of  systems  engineering  and 
developmental  testing 
requirements  that  are  aimed  at 
helping  weapon  programs  establish 
a  solid  foundation  from  the  start  of 
development. 

GAO  was  asked  to  examine 
(1)  DOD’s  progress  in 
implementing  the  systems 
engineering  and  developmental 
testing  requirements,  (2)  views  on 
the  alignment  of  the  offices  of  the 
Directors  of  Systems  Engineering 
and  Developmental  Test  and 
Evaluation,  and  (3)  challenges  in 
strengthening  systems  engineering 
and  developmental  testing 
activities.  In  conducting  this  work, 
GAO  analyzed  implementation 
status  documentation  and  obtained 
opinions  from  current  and  former 
DOD  systems  engineering  and 
testing  officials  on  the  placement 
of  the  two  offices  as  well  as 
improvement  challenges. 


What  GAO  Recommends 


GAO  recommends  that  DOD 
develop  performance  criteria  to 
assess  program  risk;  track  the 
extent  to  which  directors’ 
recommendations  are 
implemented;  address  identified 
workforce  and  training  needs;  and 
report  to  Congress  on  the  status  of 
these  efforts.  DOD  concurred  with 
the  recommendations. 


View  GAO-10-774  or  key  components. 
For  more  information,  contact  Michael  J. 
Sullivan  at  (202)  512-4841  or 
suliivanm@gao.gov. 
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What  GAO  Found 

DOD  has  implemented  or  is  implementing  the  Reform  Act  requirements 
related  to  systems  engineering  and  developmental  testing.  Several 
foundational  steps  have  been  completed.  For  example,  new  offices  have  been 
established,  directors  have  been  appointed  for  both  offices,  and  the  directors 
have  issued  a  joint  report  that  assesses  their  respective  workforce  capabilities 
and  42  major  defense  acquisition  programs.  Many  other  requirements  that 
have  been  implemented  will  require  sustained  efforts  by  the  directors’  offices, 
such  as  approving  systems  engineering  and  developmental  testing  plans,  as 
well  as  reviewing  these  efforts  on  specific  weapon  programs.  DOD  is  studying 
the  option  of  allowing  the  Director,  Developmental  Test  and  Evaluation,  to 
serve  concurrently  as  the  Director  of  the  Test  Resource  Management  Center. 
The  directors  have  not  yet  developed  joint  guidance  for  assessing  and  tracking 
acquisition  program  performance  of  systems  engineering  and  developmental 
testing  activities.  It  is  unclear  whether  the  guidance  will  include  specific 
performance  criteria  that  address  long-standing  problems  and  program  risks, 
such  as  those  related  to  concurrency  of  development  and  production  activities 
and  adequacy  of  program  resources. 

Current  and  former  systems  engineering  and  developmental  testing  officials 
offered  varying  opinions  on  whether  the  new  directors’  offices  should  have 
been  placed  under  the  Director  of  Defense  Research  and  Engineering 
organization — an  organization  that  focuses  primarily  on  developing  and 
transitioning  technologies  to  acquisition  programs.  The  Director  of  Defense 
Research  and  Engineering  believes  aligning  the  offices  under  his  organization 
helps  address  congressional  and  DOD  desires  to  increase  emphasis  on  and 
strengthen  activities  prior  to  the  start  of  a  new  acquisition  program.  Most  of 
the  officials  GAO  spoke  with  believe  the  two  offices  should  report  directly  to 
the  Under  Secretary  for  Acquisition,  Technology  and  Logistics  or  otherwise  be 
more  closely  aligned  with  acquisition  programs  because  most  of  their 
activities  are  related  to  weapon  programs.  They  also  believe  cultural  barriers 
and  staffing  issues  may  limit  the  effectiveness  of  the  two  offices  under  the 
current  organizational  structure.  Currently,  DOD  is  not  reporting  to  Congress 
on  how  successfully  the  directors  are  effecting  program  changes,  making  it 
difficult  to  determine  if  the  current  placement  of  the  offices  makes  sense  or  if 
the  Reform  Act  is  having  an  impact. 

The  military  services  face  a  number  of  challenges  as  they  try  to  strengthen 
systems  engineering  and  developmental  testing  activities  on  acquisition 
programs.  Although  the  services  believe  they  have  enough  staff  to  perform 
both  of  these  activities,  they  have  not  been  able  to  clearly  identify  the  number 
of  staff  that  are  actually  involved.  The  Director  of  Developmental  Test  and 
Evaluation  does  not  believe  the  military  services  have  enough  testing 
personnel  and  is  concerned  that  DOD  does  not  have  the  capacity  to  train  the 
large  influx  of  contractors  that  are  expected  to  be  converted  to  government 
employees. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


July  29,  2010 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

For  years,  GAO  has  reported  on  significant  cost  overruns  on  the 
Department  of  Defense’s  (DOD)  major  weapon  system  acquisition 
programs.  Even  though  DOD  has  incorporated  previous  legislative 
provisions  into  its  acquisition  policies,  such  as  requiring  weapon  programs 
to  use  mature  technologies  from  the  start  of  development,  programs  are 
still  experiencing  cost  and  schedule  problems.  The  Senate  Armed  Services 
Committee  reported  that  since  the  beginning  of  2006,  nearly  half  of  DOD’s 
largest  acquisition  programs  have  exceeded  Nunn-McCurdy1  cost-growth 
standards  established  by  Congress.  DOD  is  now  faced  with  making  tough 
decisions  about  the  viability  of  some  of  its  weapon  system  programs.  In 
2009,  for  example,  the  Secretary  of  Defense  proposed  canceling  or 
significantly  curtailing  weapon  programs  with  a  projected  cost  of  at  least 
$126  billion. 

Cost  and  schedule  overruns  can  be  attributed  to  a  number  of  factors  that 
occur  early  in  an  acquisition,  including  poorly  analyzed  requirements, 
design  instability,  and  inadequate  systems  engineering  and  testing.  In  May 
2009,  Congress  passed  the  Weapon  Systems  Acquisition  Reform  Act  of 
2009  (Reform  Act),2  aimed  at  improving  DOD’s  organization  and 
procedures  for  the  acquisition  of  major  weapon  systems.  This  legislation 
places  more  emphasis  on  activities  that  should  occur  early  in  weapon 
systems  development,  including  those  related  to  systems  engineering3  and 
developmental  testing,  in  order  to  help  establish  a  solid  program 
foundation  from  the  start  of  development.  The  Senate  Armed  Services 


1 10  U.S.C.  §  2433  establishes  the  requirement  for  unit  cost  reports.  If  certain  cost 
thresholds  are  exceeded  (known  as  unit  cost  or  Nunn-McCurdy  breaches),  DOD  is  required 
to  report  to  Congress  and,  in  certain  circumstances,  certify  the  program  to  Congress. 

2  Pub.  L.  No.  111-23. 

3  Systems  engineering  efforts  also  include  development  planning,  which  the  department 
considers  to  be  engineering  activities  prior  to  the  start  of  weapon  system  development. 
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Committee  asked  us  to  examine  (1)  DOD’s  progress  in  implementing 
systems  engineering  and  developmental  testing  requirements  called  for  in 
the  Reform  Act,  (2)  views  on  the  alignment  of  the  offices  of  the  Director  of 
Systems  Engineering  and  the  Director  of  Developmental  Test  and 
Evaluation  within  the  Office  of  the  Secretary  of  Defense,  and 
(3)  challenges  in  strengthening  systems  engineering  and  developmental 
testing  activities. 

In  conducting  our  work,  we  interviewed  officials  and  collected  documents 
from  the  offices  of  the  Director  of  Systems  Engineering  and  the  Director  of 
Developmental  Test  and  Evaluation  in  order  to  learn  the  status  of  their 
efforts  to  implement  the  Reform  Act  legislation  and  challenges  they  are 
addressing.  We  also  interviewed  officials  from  various  offices  within  the 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (AT&L);  the  office  of  the  Director,  Operational  Test  and 
Evaluation;  each  of  the  military  services;  the  Defense  Science  Board;  as 
well  as  former  DOD  systems  engineering  and  developmental  testing 
executives  to  obtain  their  opinions  on  the  alignment  of  the  two  offices 
within  the  Office  of  the  Secretary  of  Defense  and  potential  challenges.  We 
conducted  this  performance  audit  from  December  2009  to  July  2010  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 


Background 


Systems  engineering  and  test  and  evaluation  are  critical  parts  of  the 
weapon  system  acquisition  process  and  how  well  these  activities  are 
conducted  early  in  the  acquisition  cycle  can  greatly  affect  program 
outcomes.  Systems  engineering  translates  customer  needs  into  specific 
product  requirements  for  which  requisite  technological,  software, 
engineering,  and  production  capabilities  can  be  identified  through 
requirements  analysis,  design,  and  testing.  Early  systems  engineering 
provides  the  knowledge  that  weapon  system  requirements  are  achievable 
with  available  resources  such  as  technologies,  time,  people,  and  money.  It 
allows  a  product  developer  to  identify  and  resolve  performance  and 
resource  gaps  before  product  development  begins  by  reducing 
requirements,  deferring  them  to  the  future,  or  increasing  the  estimated 
cost  for  the  weapon  system’s  development.  Systems  engineering  plays  a 
fundamental  role  in  the  establishment  of  the  business  case  for  a  weapon 
acquisition  program  by  providing  information  to  DOD  officials  to  make 
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tradeoffs  between  requirements  and  resources.  Systems  engineering  is 
then  applied  throughout  the  acquisition  process  to  manage  the  engineering 
and  technical  risk  in  designing,  developing,  and  producing  a  weapon 
system.  The  systems  engineering  processes  should  be  applied  prior  to  the 
start  of  a  new  weapon  acquisition  program  and  then  continuously 
throughout  the  life-cycle. 

Test  and  evaluation  provides  information  about  the  capabilities  of  a 
weapon  system  and  can  assist  in  managing  program  risk.  There  are 
generally  two  broad  categories  of  testing:  developmental  and  operational. 
Developmental  testing  is  used  to  verify  the  status  of  technical  progress, 
substantiate  achievement  of  contract  technical  performance,  and  certify 
readiness  for  initial  operational  testing.  Early  developmental  testing 
reduces  program  risks  by  evaluating  performance  at  progressively  higher 
component  and  subsystem  levels,  thus  allowing  program  officials  to 
identify  problems  early  in  the  acquisition  process.  Developmental  testing 
officials  in  the  Office  of  the  Secretary  of  Defense  and  the  military  services 
provide  guidance  and  assistance  to  program  managers  on  how  to  develop 
sound  test  plans.  The  amount  of  developmental  testing  actually  conducted 
however,  is  controlled  by  the  program  manager  and  the  testing 
requirements  explicitly  specified  in  the  development  contract.  In  contrast, 
operational  testing  determines  if  a  weapon  system  provides  operationally 
useful  capability  to  the  warfighter.  It  involves  field  testing  a  weapon 
system,  under  realistic  conditions,  to  determine  the  effectiveness  and 
suitability4  of  the  weapon  for  use  in  combat  by  military  users,  and  the 
evaluation  of  the  results  of  such  tests.  DOD’s  Director  of  Operational  Test 
and  Evaluation  conducts  independent  assessments  of  programs  and 
reports  the  results  to  the  Secretary  of  Defense  and  Congress. 

In  2008,  the  Defense  Science  Board  reported  that  operational  testing  over 
the  previous  10  years  showed  that  there  had  been  a  dramatic  increase  in 
the  number  of  weapon  systems  that  did  not  meet  their  suitability 
requirements.  The  board  found  that  failure  rates  were  caused  by  several 
factors,  notably  the  lack  of  a  disciplined  systems  engineering  process  early 
in  development  and  a  robust  reliability  growth  program.  The  board  also 
found  that  weaknesses  in  developmental  testing,  acquisition  workforce 
reductions  and  retirements,  limited  government  oversight,  increased 


4  Effectiveness  refers  to  the  ability  of  the  system  to  perform  its  mission.  Suitability  refers  to 
the  ability  to  place  and  sustain  the  system  in  the  field.  Suitability  measures  include 
reliability,  availability,  and  logistics  supportability. 
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complexity  of  emerging  weapon  systems,  and  increased  reliance  on 
commercial  standards  (in  lieu  of  military  specifications  and  standards)  all 
contributed  to  these  failure  rates.  For  example,  over  the  last  15  years,  all 
service  acquisition  and  test  organizations  experienced  significant 
personnel  cuts,  including  the  loss  of  a  large  number  of  the  most 
experienced  technical  and  management  personnel,  including  subject 
matter  experts,  without  an  adequate  replacement  pipeline.  The  services 
now  rely  heavily  on  contractors  to  help  support  these  activities. 

Over  the  past  two  decades,  the  prominence  of  the  developmental  testing 
and  systems  engineering  communities  within  the  Office  of  the  Secretary  of 
Defense  has  continuously  evolved,  as  the  following  examples  illustrate. 

•  In  1992,  a  systems  engineering  directorate  did  not  exist  and  the 
developmental  test  function  was  part  of  the  Office  of  the  Director  of 
Test  and  Evaluation,  which  reported  directly  to  the  Under  Secretary  of 
Defense  for  Acquisition.  At  that  time,  the  director  had  direct  access  to 
the  Under  Secretary  on  an  array  of  issues  related  to  test  policy,  test 
assets,  and  the  workforce. 

•  In  1994,  the  Development  Test,  Systems  Engineering  and  Evaluation 
office  was  formed.  This  organization  effectively  expanded  the 
responsibilities  of  the  former  testing  organization  to  formally  include 
systems  engineering.  The  organization  had  two  deputy  directors:  the 
Deputy  Director,  Development  Test  and  Evaluation,  and  the  Deputy 
Director,  Systems  Engineering.  This  organization  was  dissolved  in 
1999. 

•  From  1999  to  2006,  systems  engineering  and  developmental  testing 
responsibilities  were  aligned  under  a  variety  of  offices.  The 
responsibility  for  managing  test  ranges  and  resources,  for  example, 
was  transferred  to  the  Director  of  Operational  Test  and  Evaluation. 
This  function  was  later  moved  to  the  Test  Resource  Management 
Center,  which  reports  directly  to  AT&L,  where  it  remains  today.  In 
2004,  a  Director  of  Systems  Engineering  was  re-established  and  then  in 
2006  this  became  the  System  and  Software  Engineering  Directorate. 
Developmental  testing  activities  were  part  of  this  directorate’s 
responsibilities.  As  a  result,  systems  engineering  and  developmental 
testing  issues  were  reported  indirectly  to  AT&L  through  the  Deputy 
Under  Secretary  for  Acquisition  and  Technology. 

Congress  passed  the  Weapon  Systems  Acquisition  Reform  Act  of  2009 
(Reform  Act) — the  latest  in  a  series  of  congressional  actions  taken  to 
strengthen  the  defense  acquisition  system.  The  Reform  Act  establishes  a 
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Director  of  Systems  Engineering  and  a  Director  of  Developmental  Test  and 
Evaluation  within  the  Office  of  the  Secretary  of  Defense  and  defines  the 
responsibilities  of  both  offices.  The  Reform  Act  requires  the  services  to 
develop,  implement,  and  report  on  their  plans  for  ensuring  that  systems 
engineering  and  developmental  testing  functions  are  adequately  staffed  to 
meet  the  Reform  Act  requirements.  In  addition,  it  requires  the  directors  to 
report  to  Congress  on  March  31  of  each  year  on  military  service  and  major 
defense  acquisition  program  systems  engineering  and  developmental 
testing  activities  from  the  previous  year.  For  example,  the  report  is  to 
include  a  discussion  of  the  extent  to  which  major  defense  acquisition 
programs  are  fulfilling  the  objectives  of  their  systems  engineering  and 
developmental  test  and  evaluation  master  plans,  as  well  as  provide  an 
assessment  of  the  department’s  organization  and  capabilities  to  perform 
these  activities.  Figure  1  shows  some  of  the  major  reorganizations  over  the 
past  two  decades,  including  the  most  recent  change  where  DOD  decided 
to  place  the  two  new  directors’  offices  under  the  Director  of  Defense 
Research  and  Engineering. 
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Figure  1 :  Major  Changes  in  Organizational  Placement  of  Systems  Engineering  and  Developmental  Testing  Activities  within 
the  Office  of  the  Secretary  of  Defense 


1992a  2006  2009 


Developmental  Test  Si  Evaluation  Activities 
Systems  Engineering  Activities 


Source:  GAO  presentation  of  Defense  Science  Board  and  DOD  information. 

’There  was  no  systems  engineering  office  within  the  Office  of  the  Secretary  of  Defense  in  1992.  DOD 
established  a  combined  developmental  testing  and  systems  engineering  office  in  1994. 

"Director,  Test  and  Evaluation,  had  oversight  responsibilities  for  developmental  and  live-fire  testing, 
weapon  system  assessments,  and  test  facilities  and  resources. 


DOD  Has  Made 
Progress  in 
Implementing  Reform 
Act  Requirements,  but 
Has  Not  Developed 
Performance  Criteria 
to  Track  Success 


DOD  has  made  progress  in  implementing  the  systems  engineering  and 
developmental  test  and  evaluation  provisions  of  the  Reform  Act,  but  has 
not  yet  developed  performance  criteria  that  would  help  assess  the 
effectiveness  of  the  changes.  Some  requirements,  such  as  the 
establishment  of  the  two  new  offices,  have  been  fully  implemented.  The 
implementation  of  other  requirements,  such  as  the  review  and  approval  of 
systems  engineering  and  developmental  test  and  evaluation  plans,  has 
begun  but  requires  sustained  efforts.  The  department  has  not  fully 
implemented  other  requirements.  For  example,  DOD  has  begun 
development  of  joint  guidance  that  will  identify  measurable  performance 
criteria  to  be  included  in  the  systems  engineering  and  developmental 


Page  6 


GAO-10-774  Reform  Act 


testing  plans.  DOD  initially  decided  that  one  discretionary  provision  of  the 
act — naming  the  Director  of  Developmental  Test  and  Evaluation  also  as 
the  Director  of  the  Test  Resource  Management  Center — would  not  be 
implemented.  However,  the  Director  of  Defense  Research  and  Engineering 
is  currently  examining  the  implications  of  this  organizational  change.  It 
will  be  several  years  before  the  full  impact  of  the  Reform  Act  provisions  is 
known. 

The  offices  of  the  Director  of  Systems  Engineering  and  Developmental 
Test  and  Evaluation  were  officially  established  by  the  Under  Secretary  of 
Defense  for  AT&L  in  June  2009  to  be  his  principal  advisors  on  systems 
engineering  and  developmental  testing  matters.  The  directors  took  office 
3  months  and  9  months  later,  respectively,  and  are  working  on  obtaining 
the  funding,  workforce,  and  office  space  needed  to  accomplish  their 
responsibilities.  The  directors  have  also  completed  evaluations  of  the 
military  services’  organizations  and  capabilities  for  conducting  systems 
engineering  and  developmental  testing,  and  identified  areas  for 
improvement.5  These  evaluations  were  based  on  reports  provided  by  the 
services  that  were  also  required  by  the  Reform  Act.6 

As  shown  in  table  1,  many  of  the  requirements  that  have  been 
implemented  will  require  ongoing  efforts. 


5  The  Reform  Act  requires  that  the  Director  of  Developmental  Test  and  Evaluation  and  the 
Director  of  Systems  Engineering  issue  a  joint  annual  report  not  later  than  March  31  each 
year,  beginning  in  2010,  to  the  congressional  defense  committees  addressing  activities 
undertaken  to  meet  various  requirements  of  the  Reform  Act.  Pub.  L.  No.  111-23,  §  102(a) 
(codified  at  10  U.S.C.  §  139d). 

6  The  Reform  Act  requires  that  the  service  acquisition  executive  of  each  military 
department  and  each  defense  agency  with  responsibility  for  a  major  defense  acquisition 
program  submit  a  report  to  the  Director  of  Developmental  Test  and  Evaluation  and  the 
Director  of  Systems  Engineering  on  the  status  of  the  development  and  implementation  of 
their  plans  for  ensuring  that  developmental  testing  and  system  engineering  functions  are 
adequately  staffed.  Pub.  L.  No.  111-23,  §  102(b). 


Page  7 


GAO-10-774  Reform  Act 


Table  1:  Implementation  Status  of  Key  Reform  Act  Provisions  Related  to  Systems  Engineering  and  Developmental  Testing 

Reform  Act  provision 

Systems  engineering 

Developmental  testing 

Establish  office,  appoint  director 

Completed;  ongoing  efforts  to  obtain 
needed  staff,  budget,  and  office  space 

Completed;  ongoing  efforts  to  obtain  needed 
staff,  budget,  and  office  space 

Act  as  principal  advisor  to  AT&L  and 
subject  to  the  supervision  of  AT&L 

Ongoing  efforts;  reports  indirectly  to  AT&L 
through  the  Director,  Defense  Research 
and  Engineering,  on  major  defense 
acquisition  programs 

Ongoing  efforts;  reports  indirectly  to  AT&L 
through  the  Director,  Defense  Research  and 
Engineering,  on  major  defense  acquisition 
programs 

Directors  should  coordinate  closely  to  fully 
integrate  developmental  testing  and 
systems  engineering  activities  in  DOD 

Ongoing  effort 

Ongoing  effort 

Develop  policies  and  guidance 

Ongoing  effort.  In  fiscal  year  2009, 
published  new  policy  that  expands 
reliability,  availability,  and  maintainability 
guidance  for  acquisition  programs  and 
updated  the  Defense  Acquisition 
Guidebook  c hapteron  systems 
engineering.  Also,  updating  systems 
engineering  plan  guidance  (to  be  released 
in  2010)  and  the  Guide  for  Integrating 
Systems  Engineering  into  DOD 

Acquisition  Contracts  (to  be  released  in 
fiscal  year  2011). 

O  ngoing  effort;  in  fiscal  year  2009,  published 
guidance  on  incorporating  test  and 
evaluation  requirements  into  acquisition 
contracts.  Updated  required  content  in  test 
and  evaluation  strategy  and  master  plan 
documents  to  include  reliability  factors. 

Review,  approve  acquisition  planning 
documents 

O  ngoing  effort;  in  fiscal  year  2009 
reviewed  22  and  approved  16  systems 
engineering  plans 

Ongoing  effort;  in  fiscal  year  2009  reviewed 
and  approved  25  developmental  test  and 
evaluation  plans 

Monitor,  review  activities  of  major 
acquisition  programs 

O  ngoing  effort;  in  fiscal  year  2009, 
reviewed  systems  engineering  activities 
on  35  programs.  In  2009,  participated  in 

20  technical  reviews 

Ongoing  effort;  in  fiscal  year  2009  reviewed 
developmental  testing  activities  on  17 
programs 

Provide  advocacy,  oversight,  and  guidance 
for  respective  DOD  acquisition  workforce 
career  fields 

Ongoing  effort;  acts  as  the  principal 
leader  in  DOD  for  over  45,000  people  in 
two  engineering  career  fields. 

Assessment  of  systems  engineering 
competencies  is  underway. 

Ongoing  effort;  acts  as  DOD's  principal 
leader  for  over  7,000  people  in  the  test  and 
evaluation  acquisition  career  field.  In  fiscal 
years  2009  and  2010  updated  education  and 
training  requirements  and  validated 
certification  requirements. 

Review  military  services  organizations  and 
capabilities;  identify  needed  changes  or 
improvements 

Ongoing  effort;  completed  evaluation  of 
service  reports  and  identified  weakness  in 
staffing  levels  and  expertise 

Ongoing  effort;  completed  evaluation  of 
service  reports  and  identified  weakness  in 
staffing  levels  and  expertise 

Director  of  Developmental  Test  and 
Evaluation  may  serve  as  Director  of  the 
TestResource  Management  Center 

Notapplicable 

Discretionary  provision  not  exercised  initially; 
however,  decision  is  being  reexamined 

Prepare  joint  annual  report  to  Congress 

Ongoing  effort.  First  report  issued  on  March  31,  2010.  Future  reports  are  required  by 
March  31  each  year. 
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Reform  Act  provision 

Systems  engineering  Developmental  testing 

Issue  joint  guidance  on 

•  the  development  and  tracking  of 
performance  criteria 

•  use  of  developmental  test  and 
evaluation  to  measure  achievement  of 
performance  objectives 

•  a  system  to  store  and  track 
achievement  of  performance  criteria 
and  objectives 

Not  yet  completed;  efforts  underway  to  develop  criteria 

Source:  GAO  presentation  of  DOD  data. 


The  directors  have  the  responsibility  for  reviewing  and  approving  systems 
engineering  and  developmental  test  and  evaluation  plans  as  well  as  the 
ongoing  responsibility  to  monitor  the  systems  engineering  and 
developmental  test  and  evaluation  activities  of  major  defense  acquisition 
programs.  During  fiscal  year  2009,  the  Director  of  Systems  Engineering 
reviewed  22  systems  engineering  plans  and  approved  16,  while  the 
Director  of  Developmental  Test  and  Evaluation  reviewed  and  approved  25 
developmental  test  and  evaluation  plans  within  the  test  and  evaluation 
master  plans.  Both  offices  are  monitoring  and  reviewing  activities  on  a 
number  of  major  acquisition  programs,  including  the  Virginia  Class 
Submarine,  the  Stryker  Family  of  Vehicles,  and  the  C-130  Avionics 
Modernization  Program.  Once  their  offices  are  fully  staffed,  the  directors 
plan  to  increase  efforts  in  reviewing  and  approving  applicable  planning 
documents  and  monitoring  the  activities  of  about  200  major  defense 
acquisition  and  information  system  programs. 

Evaluations  of  42  weapon  systems7  were  included  in  the  directors’  first 
annual  joint  report  to  Congress.  The  individual  systems  engineering 
program  assessments  were  consistent  in  that  they  typically  included 
information  on  10  areas,  including  requirements,  critical  technologies, 
technical  risks,  reliability,  integration,  and  manufacturing.  In  some  cases, 
the  assessments  also  included  an  overall  evaluation  of  whether  the 
program  was  low,  medium,  or  high  risk;  the  reasons  why;  and  a  general 
discussion  of  recommendations  or  efforts  the  director  has  made  to  help 
program  officials  reduce  any  identified  risk.  Examples  include  the 
following. 


'  Depending  on  a  weapon  system’s  activity  for  the  year,  an  assessment  may  include  a 
summary  of  only  developmental  testing  activity,  systems  engineering  activity,  or  both. 
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•  In  an  operational  test  readiness  assessment  of  the  EA-18G  aircraft,  the 
Director  of  Systems  Engineering  found  multiple  moderate-level  risks 
related  to  software,  communications,  and  mission  planning  and  made 
recommendations  to  reduce  the  risks.  The  program  acted  on  the  risks 
and  recommendations  identified  in  the  assessment  and  delayed  the 
start  of  initial  operational  testing  by  6  weeks  to  implement  the  fixes.  It 
has  completed  initial  operational  testing  and  was  found  to  be  effective 
and  suitable  by  Navy  testers.  The  Director  of  Operational  Test  and 
Evaluation  rated  the  system  effective  but  not  suitable,  and  stated  that 
follow-on  testing  has  been  scheduled  to  verify  correction  of  noted 
deficiencies.  The  program  received  approval  to  enter  full  rate 
production  and  is  rated  as  a  low  risk  in  the  joint  annual  report. 

•  The  systems  engineering  assessment  of  the  Global  Hawk  program  was 
high  risk  pending  the  determination  of  actual  system  capability;  it  also 
stated  that  there  is  a  high  probability  that  the  system  will  fail 
operational  testing.  The  assessment  cited  numerous  issues,  including 
questions  regarding  the  system’s  ability  to  meet  mission  reliability 
requirements,  poor  system  availability,  and  the  impact  of  simultaneous 
weapon  system  block  builds  (concurrency).  Despite  the  director’s 
concerns  and  efforts  to  help  the  program  office  develop  a  reliability 
growth  plan  for  Global  Hawk,  no  program  funding  has  been  allocated 
to  support  reliability  improvements. 

•  The  Expeditionary  Fighting  Vehicle  assessment  did  not  include  an 
overall  evaluation  of  risk.  The  assessment  noted  that  the  program  was 
on  track  to  meet  the  reliability  key  performance  parameter  of 

43.5  hours  mean  time  between  operational  mission  failure.  Problems 
related  to  meeting  this  and  other  reliability  requirements  were  a 
primary  reason  why  the  program  was  restructured  in  2007.  However, 
the  assessment  did  not  address  the  high  degree  of  concurrency 
between  development  and  production,  which  will  result  in  a 
commitment  to  fund  96  low-rate  initial  procurement  vehicles  prior  to 
demonstrating  that  the  vehicle  can  meet  the  reliability  threshold  value 
at  initial  operational  test  and  evaluation,  currently  scheduled  for 
completion  by  September  2016.8 

Developmental  testing  assessments  covered  fewer  programs  and  were  not 
as  structured  as  those  provided  by  the  systems  engineering  office  in  that 
there  were  no  standard  categories  of  information  that  were  included  in 


8  GAO,  Expeditionary  Fighting  Vehicle  (EFV)  Program  Faces  Cost,  Schedule  and 
Performance  Risks,  GAO-10-758R  (Washington,  D.C.:  July  2,  2010). 
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each  assessment.  Part  of  the  reason  is  that  the  Director  of  the 
Developmental  Test  and  Evaluation  office  was  just  developing  the 
necessary  expertise  to  review  and  provide  formal  assessments  of 
programs.  For  the  programs  that  were  reviewed,  the  assessments  included 
a  status  of  developmental  testing  activities  on  programs  and  in  some  cases 
an  assessment  of  whether  the  program  was  low,  medium,  or  high  risk.  For 
example,  the  Director  of  Developmental  Test  and  Evaluation  supported  an 
assessment  of  operational  test  readiness  for  the  C-5  Reliability 
Enhancement  and  Reengining  Program.  The  assessment  stated  that  due  to 
incomplete  testing  and  technical  issues  found  in  developmental  testing, 
there  is  a  high  risk  of  failure  in  operational  testing.  The  assessment 
recommended  that  the  program  resolve  these  issues  before  beginning 
operational  testing. 

The  Reform  Act  also  requires  that  the  Director  of  Systems  Engineering 
develop  policies  and  guidance  on,  among  other  things,  the  use  of  systems 
engineering  principles  and  best  practices  and  the  Director  of 
Developmental  Test  and  Evaluation  develop  policies  and  guidance  on, 
among  other  things,  the  conduct  of  developmental  testing  within  DOD.9 
The  directors  have  issued  some  additional  policies  to  date,  such  as 
expanded  guidance  on  addressing  reliability  and  availability  on  weapon 
programs  and  on  incorporating  test  requirements  in  acquisition  contracts. 
The  directors  plan  to  update  current  guidance  and  issue  additional 
guidance  in  the  future.  According  to  DOD  officials,  there  are  over  25 
existing  documents  that  provide  policy  and  guidance  for  systems 
engineering  and  developmental  testing.  The  directors  also  have  an  ongoing 
responsibility  to  advocate  for  and  support  their  respective  DOD 
acquisition  workforce  career  fields,  and  have  begun  examining  the  training 
and  education  needs  of  these  workforces. 

Two  provisions,  one  of  which  is  discretionary,  have  not  been  completed. 
The  Reform  Act  requires  that  the  directors,  in  coordination  with  the  newly 
established  office  of  the  Director  for  Program  Assessments  and  Root 
Cause  Analysis,  issue  joint  guidance  on  the  development  of  detailed, 
measurable  performance  criteria  that  major  acquisition  programs  should 
include  in  their  systems  engineering  and  testing  plans.  The  performance 
criteria  would  be  used  to  track  and  measure  the  achievement  of  specific 
performance  objectives  for  these  programs,  giving  decision  makers  a 
clearer  understanding  each  program’s  performance  and  progress.  The 


9  Pub.  L.  No.  111-23,  §  102(a)  (codified  at  10  U.S.C.  §  139d). 
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offices  have  begun  efforts  to  develop  these  policies  and  guidance,  but 
specific  completion  dates  have  not  been  identified.  At  this  time,  it  is 
unclear  whether  the  guidance  will  include  specific  performance  criteria 
that  should  be  consistently  tracked  on  programs  and  any  risks  associated 
with  these  programs,  such  as  ones  related  to  technology  maturity,  design 
stability,  manufacturing  readiness,  concurrency  of  development  and 
production  activities,  prototyping,  and  adequacy  of  program  resources. 
Finally,  the  Reform  Act  gives  DOD  the  option  of  permitting  the  Director  of 
Developmental  Test  and  Evaluation  to  serve  as  the  Director  of  the  Test 
Resource  Management  Center.  DOD  initially  decided  not  to  exercise  this 
option.  However,  the  Director  of  Defense  Research  and  Engineering 
recently  stated  that  his  organization  is  examining  the  possibility  of 
consolidating  the  offices.  The  director  stated  that  it  makes  sense  to 
combine  the  two  offices  because  it  would  merge  test  oversight  and  test 
resource  responsibilities  under  one  organization,  but  the  ultimate  decision 
will  be  based  on  whether  there  are  any  legal  obstacles  to  combining  the 
two  offices. 

While  most  of  the  Reform  Act’s  requirements  focus  on  activities  within  the 
Office  of  the  Secretary  of  Defense,  the  military  services  are  ultimately 
responsible  for  ensuring  that  their  weapon  systems  start  off  with  strong 
foundations.  To  that  end,  in  November  2009,  the  services,  in  reports  to  the 
Directors  of  Systems  Engineering  and  Developmental  Test  and  Evaluation, 
identified  plans  for  ensuring  that  appropriate  resources  are  available  for 
conducting  systems  engineering  and  developmental  testing  activities.  The 
individual  reports  also  highlighted  management  initiatives  undertaken  to 
strengthen  early  weapon  acquisition  activities.  For  example,  the  Army  is 
establishing  a  center  at  Aberdeen  Proving  Ground  that  will  focus  on 
improving  reliability  growth  guidance,  standards,  methods,  and  training 
for  Army  acquisition  programs.  The  Navy  has  developed  criteria,  including 
major  milestone  reviews  and  other  gate  reviews,  to  assess  the  “health”  of 
testing  and  evaluation  at  various  points  in  the  acquisition  process.  The  Air 
Force  has  undertaken  an  initiative  to  strengthen  requirements  setting, 
systems  engineering,  and  developmental  testing  activities  prior  to  the  start 
of  a  new  acquisition  program.  Air  Force  officials  believe  this  particular 
initiative  will  meet  the  development  planning  requirements  of  the  Reform 
Act. 
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Experts  Offer  Varying 
Opinions  on  the 
Placement  of  the 
Systems  Engineering 
and  Developmental 
Test  and  Evaluation 
Offices 


Experts  provided  different  viewpoints  on  the  proper  placement  of  the  new 
systems  engineering  and  developmental  test  and  evaluation  offices,  with 
some  expressing  concern  that  as  currently  placed,  the  offices  will  wield 
little  more  power  or  influence  than  they  had  prior  to  the  passage  of  the 
Reform  Act.  According  to  the  Director  of  Defense  Research  and 
Engineering,  the  Under  Secretary  of  Defense  for  AT&L  placed  the  new 
offices  under  his  organization  because  the  department  wanted  to  put 
additional  emphasis  on  systems  engineering  and  developmental  testing 
prior  to  the  start  of  a  weapons  acquisition  program.  The  director  believes 
this  is  already  occurring  and  that  both  offices  will  continue  to  have  a 
strong  relationship  with  acquisition  programs  even  though  they  do  not 
report  directly  to  an  organization  with  significant  involvement  with  major 
defense  acquisition  programs.  However,  many  current  and  former  DOD 
systems  engineering  and  developmental  testing  officials  we  spoke  with 
believe  the  offices  should  be  closely  linked  to  weapon  acquisition 
programs  because  most  of  their  activities  are  related  to  those  programs. 
Similarly,  the  Defense  Science  Board  recommended  that  a  developmental 
testing  office  be  established  and  report  directly  to  an  organization  that  has 
significant  involvement  with  major  defense  acquisition  programs.  In 
addition,  officials  we  spoke  with  believe  several  other  significant 
challenges,  including  those  related  to  staffing  and  the  culture  of  the 
Defense  Research  and  Engineering  organization,  are  already  negatively 
affecting  the  offices’  effectiveness.  DOD  has  not  established  any 
performance  criteria  that  would  help  gauge  the  success  of  the  new 
directors’  offices,  making  it  difficult  to  determine  if  the  offices  are 
properly  aligned  within  the  department  or  if  the  Reform  Act  is  having  an 
impact  on  program  outcomes. 
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DOD  Aligned  New  Systems 
Engineering  and 
Developmental  Test  and 
Evaluation  Offices  with  the 
Research  and  Engineering 
Organization 


After  the  passage  of  the  Reform  Act,  DOD  considered  several  options  on 
where  to  place  the  new  offices  of  the  Director  of  Systems  Engineering  and 
Director  of  Developmental  Test  and  Evaluation.  According  to  an  official 
who  helped  evaluate  potential  alternatives,  DOD  could  have  aligned  the 
offices  under  AT&L  in  several  different  ways  (see  fig.  2).  For  example,  the 
offices  could  have  reported  directly  to  the  Under  Secretary  of  AT&L  or 
indirectly  to  the  Under  Secretary  of  AT&L  either  through  the  Assistant 
Secretary  of  Defense  (Acquisition) 10  or  the  Director  of  Defense  Research 
and  Engineering.  DOD  decided  to  place  the  offices  under  the  Director  of 
Defense  Research  and  Engineering,  an  organization  that  previously 
primarily  focused  on  science  and  technology  issues. 


10  The  National  Defense  Authorization  Act  for  Fiscal  Year  2010  realigned  the  organizational 
structure  of  the  Office  of  the  Secretary  of  Defense.  The  effect  of  the  realignment  is  that  the 
position  of  Deputy  Under  Secretary  of  Defense  for  Acquisition  &  Technology  is  replaced  by 
the  position  of  Principal  Deputy  Under  Secretary  of  Defense  for  Acquisition,  Technology  & 
Logistics.  Also,  additional  Assistant  Secretaries  were  added,  including  the  position  of 
Assistant  Secretary  of  Defense  for  Acquisition.  Pub.  L.  No.  111-84,  §  906(a),  (b)(2)  (codified 
at  10  U.S.C.  §  137a,  138). 
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Figure  2:  Options  for  Placement  of  Directors’  Offices  for  Systems  Engineering  and  Developmental  Test  and  Evaluation 


Option  1-  Direct  report  to  USD  AT&L 

.  Option  2-  Indirect  report  to  USD  AT&L 

—  —  -  Option  3- Indirect  report  to  USD  AT&L 


Source:  DOD  data:  GAO  analysis  and  presentation. 


Director  of  Defense 
Research  and  Engineering 
Believes  Offices  Are 
Properly  Aligned 


The  Director  of  Defense  Research  and  Engineering  is  aware  of  the 
challenges  of  placing  the  offices  under  an  organization  whose  primary 
mission  is  to  develop  and  transition  technologies  to  acquisition  programs, 
but  believes  that  the  current  placement  makes  sense  given  congressional 
and  DOD  desires  to  place  more  emphasis  on  activities  prior  to  the  start  of 
a  new  acquisition  program.  He  stated  that  the  addition  of  systems 
engineering  and  developmental  testing  not  only  stretches  the  role  and 
mission  of  his  organization,  but  also  strengthens  the  organization’s  role  in 
acquisitions  because  it  helps  give  the  organization’s  research  staff  another 
point  of  view  in  thinking  about  future  technologies  and  systems. 


He  plans  for  the  offices  to  perform  both  assessment  and  advisory 
activities,  including: 
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•  providing  risk  assessments  of  acquisition  programs  for  the  Defense 
Acquisition  Board, 

•  continuing  to  help  programs  succeed  by  providing  technical  insight 
and  assisting  the  programs  in  the  development  of  the  systems 
engineering  plan  and  the  test  and  evaluation  master  plan,  and 

•  educating  and  assisting  researchers  to  think  through  new  concepts  or 
technologies  using  systems  engineering  to  inform  fielding  and 
transition  strategies. 

According  to  the  Director  of  Defense  Research  and  Engineering,  the 
offices  are  already  performing  some  of  these  functions.  For  example,  the 
new  directors  have  provided  technical  input  to  the  Defense  Acquisition 
Board  on  various  weapons  programs.  The  director  stated  the  systems 
engineering  organization  is  reviewing  manufacturing  processes  and 
contractor  manufacturing  readiness  for  weapons  programs  such  as  the 
Joint  Strike  Fighter.  In  addition,  a  developmental  testing  official  stated 
they  are  assisting  the  Director  of  Defense  Research  and  Engineering 
Research  Directorate  in  conducting  technology  readiness  assessments  and 
helping  programs  identify  the  trade  spaces  for  testing  requirements  while 
reviewing  the  test  and  evaluation  master  plan.  The  director  believes  the 
value  of  having  the  offices  perform  both  assessment  and  advisory 
activities  is  that  they  can  look  across  the  acquisition  organization  and 
identify  programs  that  are  succeeding  from  a  cost,  schedule,  and 
performance  perspective  and  identify  common  threads  or  trends  that 
enable  a  program  to  succeed.  Conversely,  they  could  identify  common 
factors  that  make  programs  fail. 

The  Director  of  Defense  Research  and  Engineering  identified  three 
challenges  that  he  is  trying  to  address  in  order  for  systems  engineering  and 
developmental  testing  to  have  a  more  positive  influence  on  weapon  system 
outcomes.  First,  the  director  would  like  to  improve  the  technical  depth  of 
the  systems  engineering  and  developmental  testing  offices.  Both  functions 
have  atrophied  over  the  years  and  need  to  be  revitalized.  This  will  require 
the  offices  to  find  highly  qualified  people  to  fill  the  positions,  which  will 
not  be  easy.  Second,  the  director  wants  to  improve  the  way  the  Defense 
Research  and  Engineering  organization  engages  with  other  DOD 
organizations  that  are  involved  in  weapon  system  acquisition.  The  director 
noted  that  there  are  a  lot  of  players  and  processes  involved  in  weapon 
acquisition  and  that  the  systems  engineering  office  can  play  a  large  role  in 
facilitating  greater  interaction.  Third,  the  director  would  like  the  Defense 
Research  and  Engineering  organization  to  find  better  ways  to  shape, 
engage  with,  contract  with,  and  get  information  from  the  defense 
industrial  base. 
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In  addition  to  the  three  challenges,  it  will  also  be  difficult  to  determine 
whether  the  two  new  offices  are  having  a  positive  impact  on  weapon 
system  outcomes.  The  Directors  of  Systems  Engineering  and 
Developmental  Test  and  Evaluation  are  not  reporting  the  number  of 
recommendations  implemented  by  program  managers  or  the  impact  the 
recommendations  have  had  on  weapon  programs,  which  would  allow 
senior  leaders  to  gauge  the  success  of  the  two  offices.  This  type  of 
information  could  help  the  Under  Secretary  of  AT&L  determine  if  the 
offices  need  to  be  placed  under  a  different  organization,  if  the  offices  need 
to  place  more  emphasis  on  advisory  or  assessment  activities,  and  if  the 
Reform  Act  is  having  an  impact  on  program  outcomes. 


Most  Experts  Believe 
Offices  Would  Be  Better 
Aligned  under  an 
Acquisition  Organization 


The  vast  majority  of  current  and  former  DOD  systems  engineering  and  test 
officials  we  spoke  with  were  opposed  to  the  placement  of  the  offices 
under  the  Director  of  Defense  Research  and  Engineering.  Their  chief 
concern  is  that  the  mission  of  the  Director  of  Defense  Research  and 
Engineering  organization  is  primarily  focused  on  developing  new 
technologies  and  transitioning  those  technologies  to  acquisition  programs. 
While  they  recognize  that  the  systems  engineering  and  developmental 
testing  offices  need  to  be  involved  in  activities  prior  to  the  official  start  of 
a  new  weapons  program,  they  believe  the  offices’  expertise  should  be 
focused  on  helping  DOD  acquisition  programs  establish  doable 
requirements  given  the  current  state  of  technologies,  not  on  the 
technologies  themselves.  Therefore,  they  believe  the  offices  would  be 
more  appropriately  placed  under  the  newly  established  offices  of  the 
Principal  Deputy  Under  Secretary  of  Defense  for  AT&L  or  the  Assistant 
Secretary  of  Defense  for  Acquisition,  whose  missions  are  more  closely 
aligned  with  acquisition  programs. 


Some  officials  we  spoke  with  believe  that  a  cultural  change  involving  the 
focus  and  emphasis  of  the  office  of  the  Director  of  Defense  Research  and 
Engineering  will  have  to  take  place  in  order  for  that  organization  to  fully 
support  its  role  in  overseeing  acquisition  programs  and  improving  the 
prominence  of  the  two  new  offices  within  the  department.  However,  these 
same  officials  believe  that  this  cultural  change  is  not  likely  to  occur  and 
that  the  Director  of  Defense  Research  and  Engineering  will  continue  to 
focus  primarily  on  developing  and  transitioning  new  technologies  to 
weapon  programs.  Therefore,  the  offices  may  not  get  sufficient  support 
and  resources  or  have  the  clout  within  DOD  to  effect  change.  One  former 
systems  engineering  official  pointed  out  that  the  historic  association  of 
systems  engineering  with  the  Director  of  Defense  Research  and 
Engineering  does  not  bode  well  for  the  systems  engineering  office.  Based 
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upon  his  experience,  the  Director  of  Defense  Research  and  Engineering’s 
focus  and  priorities  resulted  in  a  fundamental  change  in  philosophy  for  the 
systems  engineering  mission,  the  virtual  elimination  of  a  comprehensive 
focus  on  program  oversight  or  independent  identification  of  technical  risk, 
and  a  reduction  in  systems  engineering  resources.  In  short,  he  found  that 
the  Director  of  Defense  Research  and  Engineering  consistently  focused  on 
science  and  technology,  in  accordance  with  the  organization’s  charter, 
with  systems  engineering  being  an  afterthought. 

Likewise,  current  and  former  developmental  testing  officials  are 
concerned  about  the  Director  of  Defense  Research  and  Engineering’s 
support  for  developmental  testing  activities.  They  identified  several 
staffing  issues  that  they  believe  are  key  indicators  of  a  lack  of  support. 

•  First,  they  pointed  out  that  it  took  almost  9  months  from  the  time  the 
Director  of  Developmental  Test  and  Evaluation  office  was  established 
before  a  new  director  was  in  place  compared  to  3  months  to  place  the 
Director  of  Systems  Engineering.  If  developmental  testing  was  a 
priority,  officials  believe  that  the  Director  of  Defense  Research  and 
Engineering  should  have  filled  the  position  earlier. 

•  Second,  test  officials  believe  the  Director  of  Developmental  Test  and 
Evaluation  office  needs  to  have  about  the  same  number  of  staff  as  the 
offices  of  the  Director  of  Systems  Engineering  and  the  Director  of 
Operational  Test  and  Evaluation.  According  to  officials,  DOD  currently 
plans  to  have  about  70  people  involved  with  developmental  testing 
activities,  180  people  for  systems  engineering,  and  250  for  operational 
testing.  However,  testing  officials  believe  the  offices  should  be  roughly 
the  same  size  given  the  fact  that  developmental  testing  will  cover  the 
same  number  of  programs  as  systems  engineering  and  operational 
testing  and  that  roughly  80  percent  of  all  testing  activities  are  related 
to  developmental  tests,  with  the  remaining  20  percent  being  for 
operational  tests. 

•  Third,  even  though  the  Director  of  Developmental  Test  and  Evaluation 
expects  the  office  to  grow  to  about  70  people  by  the  end  of  fiscal  year 
2011,  currently  there  are  30  people  on  board.  The  director  believes 
there  are  a  sufficient  number  of  qualified  people  seeking  positions  and 
therefore  the  office  could  be  ramped  up  more  quickly. 

•  Finally,  the  Director  of  Developmental  Test  and  Evaluation  stated  that 
his  office  has  only  one  senior-level  executive  currently  on  staff  who 
reports  to  him  and  that  there  are  no  plans  to  hire  more  for  the 
70-person  organization.  The  director  believes  it  is  crucial  that  the 
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organization  have  more  senior-level  officials  because  of  the  clout  they 
carry  in  the  department.  The  director  believes  that  the  lack  of  an 
adequate  number  of  senior  executives  in  the  office  weakens  its  ability 
to  work  effectively  with  or  influence  decisions  made  by  other  DOD 
organizations.  Further,  officials  from  other  testing  organizations,  as 
well  as  the  systems  engineering  office,  indicated  they  have  two  or 
more  senior  executive-level  employees. 

A  May  2008  Defense  Science  Board  report,  which  was  focused  on  how 
DOD  could  rebuild  its  developmental  testing  activities,  recommended  that 
developmental  testing  be  an  independent  office  that  reports  directly  to  the 
Deputy  Under  Secretary  of  Defense  (Acquisition  and  Technology).  At  that 
time,  according  to  the  report,  there  was  no  office  within  the  Office  of  the 
Secretary  of  Defense  with  comprehensive  developmental  testing  oversight 
responsibility,  authority,  or  staff  to  coordinate  with  operational  testing.  In 
addition,  the  existing  residual  organizations  lacked  the  clout  to  provide 
development  test  guidance  and  developmental  testing  was  not  considered 
to  be  a  key  element  in  AT&L  system  acquisition  oversight.  According  to 
the  study  director,  placing  the  developmental  testing  office  under  the 
Director  of  Defense  Research  and  Engineering  does  not  adequately 
position  the  new  office  to  perform  the  oversight  of  acquisition  programs. 
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Military  Services  Face 
Workforce  and 
Resource  Challenges 
as  They  Strive  to 
Strengthen  Their 
Systems  Engineering 
and  Developmental 
Testing  Efforts 


The  military  services,  the  Directors  of  Systems  Engineering  and 
Developmental  Test  and  Evaluation,  and  we  have  identified  a  number  of 
workforce  and  resource  challenges  that  the  military  services  will  need  to 
address  to  strengthen  their  systems  engineering  and  developmental  testing 
activities.  For  example,  it  is  unclear  whether  the  services  have  enough 
people  to  perform  both  systems  engineering  and  developmental  testing 
activities.  Even  though  the  services  reported  to  the  directors  that  they 
have  enough  people,  they  do  not  have  accurate  information  on  the  number 
of  people  performing  these  activities.  The  Director  of  Developmental  Test 
and  Evaluation  disagreed  with  the  services’  assertions,  but  did  not  know 
how  many  additional  people  are  needed.  Service  officials  have  also 
expressed  concern  about  the  department’s  ability  to  train  individuals 
who  do  not  meet  requisite  certification  requirements  on  a  timely  basis11 
and  being  able  to  obtain  additional  resources  to  improve  test  facilities. 


The  military  services  were  required  by  the  Reform  Act  to  report  on  their 
plans  to  ensure  that  they  have  an  adequate  number  of  trained  systems 
engineering  and  developmental  testing  personnel  and  to  identify  additional 
authorities  or  resources  needed  to  attract,  develop,  train,  and  reward  their 
staff.  In  November  2009,  the  military  services  submitted  their  reports  to 
the  respective  directors  within  the  Office  of  the  Secretary  of  Defense  on 
their  findings.  In  general,  the  services  concluded  that  even  with  some 
recruiting  and  retention  challenges,  they  have  an  adequate  number  of 
personnel  to  conduct  both  systems  engineering  and  developmental  testing 
activities  (see  table  2  below).  According  to  service  officials,  this 
determination  was  based  on  the  fact  that  no  program  offices  identified  a 
need  for  additional  staffing  to  complete  these  activities.  The  reports  also 
stated  the  services  generally  have  sufficient  authorities  to  attract  and 
retain  their  workforce.  In  DOD’s  first  annual  joint  report  to  Congress,  the 
Director  of  Developmental  Test  and  Evaluation  did  not  agree  with  the 
military  services’  assertion  that  they  have  enough  staff  to  perform  the  full 


11  Each  acquisition,  technology,  and  logistics  (AT&L)  position,  meaning  positions 
designated  to  be  acquisition  positions  in  accordance  with  the  Defense  Acquisition 
Workforce  Improvement  Act,  has  certification-level  requirements.  When  an  individual  is 
placed  in  an  AT&L  position,  the  determination  that  the  individual  has  satisfied  appropriate 
certification  and  assignment-  specific  training  requirements,  or  a  plan  for  the  individual  to 
meet  the  requirements  within  24  months  of  placement  or  other  established  period,  shall  be 
documented.  If  an  individual  does  not  meet  position  requirements  within  established  time 
frames,  a  waiver  must  be  obtained  according  to  applicable  procedures  for  the  individual  to 
remain  in  the  position.  Department  of  Defense  Instruction  5000.66,  Operation  of  the 
Defense  Acquisition,  Technology,  and  Logistics  Workforce  Education,  Training,  and  Career 
Development  Program  paragraphs  E2. 1.3.3  and  E2.4.1.2.  (Dec.  21,  2005). 
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range  of  developmental  testing  activities.  The  director  does  not  know  how 
many  more  personnel  are  needed,  but  indicated  that  the  office  plans  to 
work  with  the  services  to  identify  additional  workforce  needs.  The 
Director  of  Systems  Engineering  agreed  with  the  services’  reports  that 
they  have  adequate  staffing  to  support  systems  engineering  activities 
required  by  current  policy.  According  to  the  director,  this  was  based  on 
the  35,000  current  personnel  identified  in  the  System  Planning,  Research 
Development,  and  Engineering  workforce — a  generic  workforce  category 
that  includes  systems  engineering  activities — as  well  as  the  services’  plans 
to  hire  over  2,500  additional  personnel  into  this  same  workforce  category 
over  the  next  several  years. 


Table  2:  Military  Service  Systems  Planning,  Research  Development,  and  Engineering  and  Developmental  Testing  Personnel 


Systems  Planning,  Research 

Development,  and  Engineering3  Developmental  testing” 


Civilian 

Military 

Total 

Civilian 

Military 

Total 

Air  Force 

5,004 

1,871 

6,875 

1,354 

1,276 

2,630 

Army 

10,107 

107 

10,214 

2,131 

11 

2,142 

Navy 

17,885 

201 

18,086 

2,381 

450 

2,831 

Total 

32,996 

2,179 

35,175 

5,866 

1,737 

7,603 

Source:  GAO  presentation  of  DOD  data. 

Note:  Developmental  testing  data  for  all  three  services  are  as  of  September  2009.  The  Air  Force, 
Army,  and  Navy  systems  engineering  data  are  as  of  J  une  2009,  September  2009,  and  November 
2009,  respectively. 

The  military  services  identified  their  systems  engineering  personnel  as  those  coded  as  Program 
Systems  Engineers  and  Systems  Engineers  (a  general  classification  for  other  types  of  engineers)  in 
the  Systems  Planning,  Research  Development,  and  Engineering  workforce  classification. 

bSome  personnel  conducting  work  in  developmental  testing  may  not  be  included  because  their  work 
is  primarily  conducted  in  another  area. 

Although  not  clearly  articulated  in  the  services’  reports,  military  service 
officials  acknowledged  that  the  personnel  data  in  their  reports  may  not  be 
entirely  accurate.  For  example,  officials  believe  the  systems  engineering 
numbers  identified  in  table  2  overstate  the  number  of  people  actually 
performing  systems  engineering  activities  because  that  particular  career 
field  classification  is  a  generic  category  that  includes  all  types  of 
engineers.  The  developmental  test  workforce  shown  in  the  table  does  not 
completely  reflect  the  number  of  people  who  actually  perform 
developmental  testing  activities  because  the  information  provided  by  the 
military  services  only  identifies  the  personnel  identified  in  the  test  and 
evaluation  career  field.  Service  officials  told  us  that  there  are  many  other 
people  performing  these  activities  who  are  identified  in  other  career  fields. 
The  Director  of  Developmental  Test  and  Evaluation  believes  these  other 
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people  may  not  be  properly  certified  and  that  in  the  case  of  contractors, 
they  do  not  possess  certifications  which  are  equivalent  to  the  certification 
requirements  of  government  personnel.  This  director  plans  to  request 
another  report  from  the  services  in  fiscal  year  2010.  This  report  will 
address  the  overall  workforce  data;  it  will  cover  current  staffing  assigned 
to  early  test  and  evaluation  activities,  training,  and  certification  concerns 
they  have  related  to  in-sourcing  staff,  rapid  acquisition  resource  plans,  and 
infrastructure  needs  for  emerging  technologies.  The  Director  of  Systems 
Engineering  does  not  intend  to  request  another  report  from  the  services. 
Nevertheless,  each  of  the  military  services  plans  to  increase  its  systems 
engineering  workforce  over  the  next  several  years.  The  exact  number  of 
personnel  is  uncertain  because  the  services’  hiring  projections  relate  to  a 
general  engineering  personnel  classification,  not  a  specific  systems 
engineering  career  field. 

The  directors  also  identified  challenges  they  believe  the  services  will  face 
in  strengthening  systems  engineering  and  developmental  testing  activities. 
The  Director  of  Systems  Engineering  pointed  out  that  the  services  need  to 
put  greater  emphasis  on  development  planning  activities,  as  called  for  by 
the  Reform  Act.  The  services  are  currently  conducting  these  activities  to 
some  extent,  but  the  director  believes  a  more  robust  and  consistent 
approach  is  needed.  The  Director  of  Developmental  Test  and  Evaluation 
highlighted  two  other  challenges  facing  the  military  services.  First,  the 
director  would  like  to  increase  the  number  of  government  employees 
performing  test  and  evaluation  activities.  The  services  experienced 
significant  personnel  cuts  in  these  areas  in  the  mid-1990s  and  has  to  rely 
on  contractors  to  perform  the  work.  DOD’s  joint  report  to  Congress  noted 
that  the  Air  Force  in  particular  relies  heavily  on  prime  contractor 
evaluations  and  that  this  approach  could  lead  to  test  results  that  are 
inaccurate,  misleading,  or  not  qualified,  resulting  in  turn,  in  premature 
fielding  decisions  since  prime  contractors  would  not  be  giving  impartial 
evaluations  of  results.  The  director  believes  there  are  a  number  of 
inherently  governmental  test  and  evaluation  functions  that  produce  a  more 
impartial  evaluation  of  results  and  that  a  desired  end  state  would  be  one 
where  there  is  an  appropriate  amount  of  government  and  contractor 
testing.  Second,  the  director  is  concerned  that  DOD  does  not  have  the 
capacity  to  train  and  certify  an  estimated  800  individuals  expected  to  be 
converted  from  contractor  to  government  employees  within  the  required 
time  frame.  While  most  of  the  contractors  are  expected  to  have  some  level 
of  training  and  experience  performing  test  activities,  they  probably  will 
not  meet  certifications  required  of  government  employees  because  they 
have  not  had  the  same  access  to  DOD  training. 
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In  addition  to  those  challenges  recognized  by  the  directors,  we  have 
identified  other  challenges  we  believe  the  services  may  face  in 
implementing  more  robust  systems  engineering  and  developmental  testing, 
including  the  following. 

•  According  to  the  military  services,  they  plan  to  meet  hiring  targets 
primarily  through  the  conversion  of  contractors  who  are  already 
performing  those  activities,  but  do  not  have  plans  in  place  to  ensure 
that  they  have  the  right  mixture  of  staff  and  expertise  both  now  and  in 
the  future.  DOD  officials  acknowledge  that  they  do  not  know  the 
demographics  of  the  contractor  workforce.  However,  they  believe 
many  contractors  are  often  retired  military  with  prior  systems 
engineering  experience.  Therefore,  while  they  may  be  able  to  meet 
short-term  needs,  there  could  be  a  challenge  in  meeting  long-term 
workforce  needs. 

•  Army  test  officials  indicated  that  they  have  experienced  a  significant 
increase  in  their  developmental  testing  workload  since  the  terrorist 
attacks  of  September  2001,  with  no  corresponding  increase  in  staffing. 
As  a  result,  personnel  at  their  test  ranges  are  working  longer  hours  and 
extra  shifts,  which  testing  officials  are  concerned  may  affect  their 
retention  rates.  Army  officials  also  indicated  that  test  ranges  are 
deteriorating  more  quickly  than  expected  and  they  may  not  have  the 
appropriate  funding  to  upgrade  and  repair  the  facilities  and 
instrumentation.  Test  personnel  are  often  operating  in  obsolete  and 
outdated  facilities  that  cannot  meet  test  requirements,  resulting  in 
safety  issues,  potential  damage  to  equipment,  and  degraded  quality  of 
life. 

•  DOD’s  increased  emphasis  on  fielding  rapid  acquisition  systems  may 
require  the  services  to  tailor  their  approach  to  systems  engineering. 
According  to  an  Air  Force  official,  efforts  that  normally  take  months  to 
complete  for  a  more  traditional  acquisition  program,  have  to  be 
completed  in  a  matter  of  weeks  for  rapid  acquisition  programs. 


Conclusions 


DOD  efforts  to  implement  Reform  Act  requirements  are  progressing,  but  it 
will  take  some  time  before  the  results  of  these  efforts  can  be  evaluated. 
Current  and  former  systems  engineering  and  developmental  testing 
officials  offer  compelling  insights  concerning  the  placement  of  the  new 
directors’  offices  under  the  Office  of  the  Director  of  Defense  Research  and 
Engineering,  but  it  is  still  too  soon  to  judge  how  effective  the  offices  will 
be  at  influencing  outcomes  on  acquisition  programs.  The  current 
placement  of  the  offices  may  present  several  challenges  that  could  hinder 
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their  ability  to  effectively  oversee  weapon  system  acquisition  programs 
and  ensure  that  risks  are  identified,  discussed,  and  addressed  prior  to  the 
start  of  a  new  program  or  the  start  of  operational  testing.  Foremost  among 
these  potential  challenges  is  the  ability  of  the  Director  of  Defense 
Research  and  Engineering  to  change  the  focus  of  the  organization  to 
effectively  assimilate  the  roles  and  missions  of  the  two  new  offices  and 
then  ensure  that  the  offices  are  properly  staffed  and  have  the  appropriate 
number  of  senior  leaders.  The  mission  of  the  office  of  the  Director  of 
Defense  Research  and  Engineering  has  been  to  develop  technology  for 
weapon  programs;  its  focus  has  not  been  to  manage  the  technical  aspects 
of  weapon  system  acquisition  programs. 

Ultimately,  the  real  proof  of  whether  an  organization  outside  of  the  major 
defense  acquisition  program  arena  can  influence  acquisition  program 
decisions  and  outcomes  should  be  based  on  results.  The  directors’  offices 
have  started  to  assess  and  report  on  the  systems  engineering  and 
developmental  testing  activities  on  some  of  the  major  defense  acquisition 
programs.  They  have  also  made  recommendations  and  worked  with 
program  officials  to  help  reduce  risks  on  programs  such  as  the  EA-18G, 
Global  Hawk,  and  the  C-5  Reliability  Enhancement  and  Reengining 
programs.  However,  guidance  on  the  development  and  tracking  of 
performance  criteria  that  would  provide  an  indication  of  how  much  risk  is 
associated  with  a  particular  weapon  system — such  as  those  related  to 
technology  maturity,  design  stability,  manufacturing  readiness, 
concurrency  of  development  and  production  activities,  prototyping,  and 
adequacy  of  program  resources — has  yet  to  be  developed.  Further,  the 
directors  are  not  reporting  to  Congress  on  the  extent  to  which  programs 
are  implementing  recommendations  and  the  impact  recommendations  are 
having  on  weapon  programs,  which  would  provide  some  insight  as  to  the 
impact  the  two  offices  are  having  on  acquisition  programs.  Although  not 
required  by  the  Reform  Act,  this  type  of  information  could  be  useful  for 
Congress  to  gauge  the  effectiveness  of  the  directors’  offices. 

The  military  services,  which  face  increasing  demands  to  develop  and  field 
more  reliable  weapon  systems  in  shorter  time  frames,  may  need  additional 
resources  and  training  to  ensure  that  adequate  developmental  testing  and 
systems  engineering  activities  are  taking  place.  However,  DOD’s  first  joint 
annual  report  to  Congress,  which  was  supposed  to  assess  the  department’s 
organization  and  capabilities  for  performing  systems  engineering  and 
developmental  testing  activities,  did  not  clearly  identify  the  workforce 
performing  these  activities,  future  workforce  needs,  or  specific  hiring 
plans.  In  addition,  DOD’s  strategy  to  provide  the  necessary  training  within 
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the  required  time  period  to  the  large  number  of  staff  it  plans  to  hire  is 
unclear.  Therefore,  workforce  and  training  gaps  are  unknown. 


Recommendations 
Executive  Action 


In  order  to  determine  the  effectiveness  of  the  newly  established  offices, 
we  recommend  that  the  Secretary  of  Defense  direct  the  Directors  of 
Systems  Engineering  and  Developmental  Test  and  Evaluation  to  take  the 
following  five  actions: 


•  Ensure  development  and  implementation  of  performance  criteria  for 
systems  engineering  plans  and  developmental  test  and  evaluation 
master  plans,  such  as  those  related  to  technology  maturity,  design 
stability,  manufacturing  readiness,  concurrency  of  development  and 
production  activities,  prototyping,  and  the  adequacy  of  program 
resources. 

•  Track  the  extent  to  which  program  offices  are  adopting  systems 
engineering  and  developmental  testing  recommendations. 

•  Work  with  the  services  to  determine  the  appropriate  number  of 
government  personnel  needed  to  perform  the  scope  of  systems 
engineering  and  developmental  testing  activities. 

•  Develop  plans  for  addressing  the  training  needs  of  the  new  hires  and 
contractors  who  are  expected  to  be  converted  to  government 
personnel. 

•  Report  to  Congress  on  the  status  of  these  efforts  in  future  joint  annual 
reports  required  by  the  Reform  Act. 


Agency  Comments 
and  Our  Evaluation 


DOD  provided  us  with  written  comments  on  a  draft  of  this  report.  DOD 
concurred  with  each  of  the  recommendations,  as  revised  in  response  to 
agency  comments.  DOD’s  comments  appear  in  appendix  I. 

Based  upon  a  discussion  with  DOD  officials  during  the  agency  comment 
period,  we  revised  the  first  recommendation.  Specifically,  instead  of 
recommending  that  the  Directors  of  Systems  Engineering  and 
Developmental  Test  and  Evaluation  develop  a  comprehensive  set  of 
performance  criteria  that  would  help  assess  program  risk,  as  stated  in  the 
draft  report,  we  now  recommend  that  the  directors  ensure  the 
development  and  implementation  of  performance  criteria  for  systems 
engineering  plans  and  developmental  test  and  evaluation  master  plans. 

The  wording  change  clarifies  the  nature  and  scope  of  performance  criteria 
covered  by  our  recommendation  and  is  consistent  with  Reform  Act 
language  that  requires  the  directors  to  develop  guidance  on  the 
development  of  detailed,  measurable  performance  criteria  that  major 
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acquisition  programs  should  include  in  their  systems  engineering  and 
developmental  testing  plans.  According  to  DOD  officials,  the  military 
services  are  then  responsible  for  developing  the  specific  criteria  that 
would  be  used  on  their  respective  programs. 

DOD  also  provided  technical  comments,  which  we  incorporated  as 
appropriate. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense,  the 
Director  of  the  Office  of  Management  and  Budget,  and  interested 
congressional  committees.  We  will  also  make  copies  available  at  no  charge 
on  the  GAO  Web  site  at  http://www.gao.gov. 

If  you  have  any  questions  about  this  report  or  need  additional  information, 
please  contact  me  at  (202)  512-4841  or  sullivanm@gao.gov.  Contact  points 
for  our  Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found 
on  the  last  page  of  this  report.  Key  contributors  to  this  report  were  Bruce 
Thomas,  Assistant  Director;  Cheryl  Andrew;  Rae  Ann  Sapp;  Megan  Hill; 
and  Kristine  Hassinger. 


Michael  J.  Sullivan 
Director 

Acquisition  and  Sourcing  Management 
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Appendix  I:  Comments  from  the  Department 
of  Defense 


DIRECTOR  OF  DEFENSE  RESEARCH  AND  ENGINEERING 

3030  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3030 

JUL  2  2  201« 


Mr.  Michael  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Sullivan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report  10-774, 
"DEFENSE  ACQUISITIONS:  DoD  Needs  to  Develop  Performance  Criteria  to  Gauge  Impact  of 
Reform  Act  Changes  and  Address  Workforce  Issues,"  dated  June  18,  2010,  (GAO  Code 
120880).  Detailed  comments  on  the  report  recommendations  are  enclosed.  Detailed  comments 
on  factual  information  within  the  body  of  the  report  have  been  forwarded  to  the  GAO  action 
officer  separately. 

The  Department  appreciates  the  opportunity  to  respond  to  your  draft  report  and  looks 
forward  to  working  with  you  as  we  continue  to  ensure  a  strong  and  capable  Defense  acquisition 
capability. 


Sincerely, 


Enclosure: 
As  stated 
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Appendix  I:  Comments  from  the  Department 
of  Defense 


GAO  DRAFT  REPORT  -  DATED  JUNE  18,  2010 
GAO  CODE  1 20880/ GAO-1 0-774 

"DEFENSE  ACQUISITIONS:  DoD  Needs  to  Develop  Performance  Criteria  to 
Gauge  Impact  of  Reform  Act  Changes  and  Address  Workforce  Issues" 


DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Directors  of  Systems  Engineering  and  Developmental  Test  and  Evaluation  to  ensure 
development  and  implementation  of  performance  criteria  for  systems  engineering  plans 
and  developmental  test  and  evaluation  master  plans,  such  as  those  related  to  technology 
maturity,  design  stability,  manufacturing  readiness,  concurrency  of  development  and 
production  activities,  prototyping,  and  the  adequacy  of  program  resources. 


DOD  RESPONSE:  Concur.  The  Director,  Systems  Engineering  and  the  Director, 
Developmental  Test  and  Evaluation,  consistent  with  PL  1 1 1-23,  will  issue  guidance  on 
the  development  of  performance  criteria  for  systems  engineering  plans  and 
developmental  test  and  evaluation  master  plans  and  will  ensure  that  these  criteria  are  in 
fact  developed  and  implemented. 


RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Directors  of  Systems  Engineering  and  Developmental  Test  and  Evaluation  to  track 
the  extent  to  which  program  offices  are  adopting  systems  engineering  and  developmental 
testing  recommendations. 


DOD  RESPONSE:  Concur.  The  Director,  Systems  Engineering  and  the  Director, 
Developmental  Test  and  Evaluation  will  track  the  extent  to  which  program  offices  adopt 
systems  engineering  and  developmental  test  and  evaluation  recommendations.  However, 
these  will  not  be  reported  as  standalone  metrics.  The  Director,  Systems  Engineering  and 
the  Director,  Developmental  Test  and  Evaluation  will  continue  to  work  with  the  Services 
and  programs  to  understand  both  their  risk  management  approach  and  overall  program 
performance. 


RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretaiy  of  Defense  direct 
the  Directors  of  Systems  Engineering  and  Developmental  Test  and  Evaluation  to  work 
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Appendix  I:  Comments  from  the  Department 
of  Defense 


with  the  Services  to  determine  the  appropriate  number  of  Government  personnel  needed 
to  perform  the  scope  of  systems  engineering  and  developmental  testing  activities. 


POD  RESPONSE:  Concur.  The  Director,  Systems  Engineering  and  the  Director, 
Developmental  Test  and  Evaluation  will  work  with  the  Services  to  determine  the 
appropriate  number  of  Government  personnel  needed  to  perform  the  scope  of  systems 
engineering  and  developmental  testing  activities. 


RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Directors  of  Systems  Engineering  and  Developmental  Test  and  Evaluation  to  develop 
plans  for  addressing  the  training  needs  of  the  new  hires  and  contractors  that  are  expected 
to  be  converted  to  Government  personnel. 


POD  RESPONSE:  Concur.  The  Director,  Systems  Engineering  and  the  Director, 
Developmental  Test  and  Evaluation,  consistent  with  their  Functional  Leader  roles,  will 
develop  plans  for  addressing  the  training  needs  of  their  respective  career  fields.  The 
Directors  will  work  coilaboratively  with  the  Services  and  Components  who  are  ultimately 
responsible  for  training  and  quality  of  their  personnel. 


RECOMMENDATION  5:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Directors  of  Systems  Engineering  and  Developmental  Test  and  Evaluation  to  report  to 
the  Congress  on  the  status  of  these  efforts  in  future  joint  annual  reports  required  by  the 
Reform  Act. 


POP  RESPONSE:  Concur.  The  Director,  Systems  Engineering  and  the  Director, 
Developmental  Test  &  Evaluation  will  report  the  status  of  these  efforts  in  the  joint  annual 
report  required  by  the  Reform  Act. 


(120880) 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 


Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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